Implications of QRIS Design for the Distribution of Program Ratings and Linkages between Ratings and Observed Quality

This report is in the public domain. Permission to reproduce is not necessary. Overview This Brief compares three hypothetical Quality Rating and Improvement Systems (QRIS) that use different rating structures: block, points, and hybrid. Because the quality standards in the hypothetical QRIS are held relatively constant across structures, analyses can be conducted to determine how structure relates to key QRIS outcomes. Three outcomes are examined: the distribution of programs across ratings levels, the linkages of ratings with measures of observed quality, and the scores of individual quality components within each structure. Findings indicate that the distribution of ratings is significantly related to structure. Whereas fewer than one-fifth of programs achieved a Level 3 or 4 in the block structure, over 70% of programs achieved a Level 3 or 4 in the points and hybrid structures. Rating levels produced by each of the three structures were significantly correlated with observed quality as measured by the Early Childhood Environment Rating Scale-Revised (ECERS-R). However, the points structure was the only structure to produce quality levels in which observed quality was significantly different between each level. The points structure also captured the greatest range of ECERS-R scores with a 1.61 point spread between Level 1 and Level 4 compared to 0.13 and 1.14 point spreads for the block and hybrid structures respectively. Scores across rating levels in the rating structures showed different patterns for specific quality components, with some domains (Health and Safety, Assessment and Accreditation) scoring high regardless of level and structure, others (Family Partnerships) scoring relatively low and others (Teacher Qualifications and Director Qualifications) demonstrating how quality component scores can differ across structures. The analyses are limited in their application to QRIS because the data were collected with a unique sample and were not collected in the context of a " real " QRIS. Nevertheless, this Brief offers research evidence that can be useful to QRIS administrators as they weigh different design options and their potential consequences.